Image processing device, image processing method, and program

ABSTRACT

An image processing device includes a determination unit which determines a scheme of capturing a pixel value of an imaging element at a time of imaging operation of moving image contents using a feature amount which is obtained from a plurality of regions in a frame which configures the moving image contents.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application claims priority to Japanese Priority PatentApplication JP 2013-163789 filed in the Japan Patent Office on Aug. 7,2013, the entire content of which is hereby incorporated by reference.

BACKGROUND

The present application relates to an image processing device.Specifically, the present application relates to an image processingdevice which handles an image, an image processing method, and a programwhich causes the method to be executed in a computer.

In the related art, an image processing device such as an imagingapparatus which generates an image (image data) by imaging an objectsuch as a person using an image sensor has become widespread. Inaddition, as the image sensor, for example, there is a ComplementaryMetal Oxide Semiconductor (CMOS) image sensor, or a Charge CoupledDevice (CCD) image sensor. In addition, an image processing device suchas a reproducing device which reproduces an image which is generated inthis manner has been widely spread.

For example, a correction device which corrects distortion of an imagedimage (focal plane distortion) due to a focal plane phenomenon whichoccurs in the CMOS image sensor, or the like, has been proposed (forexample, refer to Japanese Unexamined Patent Application Publication No.2007-208580).

SUMMARY

In the above described related art, since a correction in a device suchas an imaging device is a target, and a vertical imaging direction ofthe CMOS image sensor is known in the device, it is possible toappropriately correct focal plane distortion of an imaged image based onthe vertical imaging direction.

Here, for example, a case in which moving image contents which arerecorded without performing a correction of focal plane distortion usingthe imaging device are moved to another reproducing device, and arereproduced in the reproducing device will be assumed. In this case, itis assumed that the reproducing device performs a correction of focalplane distortion with respect to the moving image contents, and themoving image contents which are subjected to the correction of the focalplane distortion are reproduced.

However, a case is also assumed in which a vertical imaging direction ofthe CMOS image sensor when imaging the moving image contents isdifferent. In this case, when the reproducing device performs acorrection of focal plane distortion with respect to the moving imagecontents based on a wrong vertical imaging direction, it is not possibleto appropriately correct the focal plane distortion.

In addition, for example, moving image contents which are imaged using aglobal shutter image sensor (for example, CCD image sensor) do not havefocal plane distortion in principle. For this reason, it is notnecessary to perform a correction of focal plane distortion with respectto moving image contents which are imaged using the global shutter imagesensor.

That is, when handling moving image contents, it is important toappropriately obtain imaging information such as a vertical imagingdirection, and a shutter system at a time of the imaging operation, anduse the information.

It is desirable to appropriately obtain imaging information at a time ofan imaging operation.

According to an embodiment of the present application, there is providedan image processing device which includes a determination unit whichdetermines a scheme of capturing a pixel value of an imaging element ata time of imaging operation of moving image contents using a featureamount which is obtained from a plurality of regions in a frame whichconfigures the moving image contents, an image processing method, and aprogram which causes the method to be executed in a computer. In thismanner, it is possible to determine the capturing scheme of the pixelvalue of the imaging element when imaging the moving image contentsusing the feature amount which is obtained from the plurality of regionsin the frame which configures the moving image contents.

In the image processing device, the determination unit may determine atleast one of an imaging direction and a shutter system of the imagingelement at the time of the imaging operation, as the capturing scheme.In this manner, it is possible to determine at least one of the imagingdirection and the shutter system of the imaging element at the time ofimaging operation, as the capturing scheme.

In the image processing device, the determination unit may make thedetermination based on continuity of local movement vectors betweenframes which are neighboring time sequentially among frames whichconfigure the moving image contents, by obtaining the local movementvector in each of the plurality of regions as the feature amount. Inthis manner, it is possible to determine the capturing scheme of thepixel value of the imaging element at the time of the imaging operationof the moving image contents based on the continuity of the localmovement vectors between the frames which are neighboring timesequentially, among the frames which configure the moving imagecontents.

In the image processing device, the determination unit may obtain alocal movement vector for each of the plurality of regions as thefeature amount, may set frames which are neighboring time sequentiallyamong frames which configure the moving image contents to a first frameand a second frame, and may make the determination based on a comparisonresult of a local movement vector which is obtained from a region on oneend side in a specific direction of the first frame, and a localmovement vector which is obtained from a region on the other end side inthe specific direction of the second frame. In this manner, it ispossible to determine the capturing scheme of the pixel value of theimaging element at the time of the imaging operation of the moving imagecontents based on the comparison result of the local movement vectorwhich is obtained from the region on the one end side in the specificdirection of the first frame, and the local movement vector which isobtained from the region on the other end side in the specific directionof the second frame.

In the image processing device, the determination unit may make thedetermination using a value which is calculated based on a comparisonresult of a first movement vector and a fourth movement vector, and acomparison result of a second movement vector and a third movementvector, the first movement vector being the local movement vectorobtained from a region on the one end side of the first frame, thesecond movement vector being the local movement vector obtained from theregion on the other end side of the first frame, the third movementvector being the local movement vector obtained from a region on the oneend side of the second frame, and the fourth movement vector being thelocal movement vector obtained from a region on the other end side ofthe second frame. In this manner, it is possible to determine thecapturing scheme of the pixel value of the imaging element at the timeof the imaging operation of the moving image contents using a valuewhich is calculated based on the comparison result of the first movementvector and the fourth movement vector, and the comparison result of thesecond movement vector and the third movement vector.

In the image processing device, a correction unit which performs acorrection of focal plane distortion with respect to the moving imagecontents based on the determined capturing scheme may be furtherincluded. In this manner, it is possible to perform the correction ofthe focal plane distortion with respect to the moving image contentsbased on the determined capturing scheme.

In the image processing device, the determination unit may determine atleast one of the imaging direction and the shutter system of the imagingelement at the time of the imaging operation as the capturing scheme,and the correction unit may perform the correction of the focal planedistortion with respect to the moving image contents based on thedetermined imaging direction when it is determined that the shuttersystem is a focal plane shutter system by the determination unit, andmay not perform the correction of the focal plane distortion withrespect to the moving image contents when it is determined that theshutter system is a global shutter system by the determination unit. Inthis manner, when it is determined that the shutter system is the focalplane shutter system, the correction of the focal plane distortion isperformed with respect to the moving image contents based on thedetermined imaging direction, and when it is determined that the shuttersystem is the global shutter system, the correction of the focal planedistortion is not performed with respect to the moving image contents.

In the image processing device, the determination unit may obtain afeature amount in each of the plurality of regions based on a comparisonresult of a plurality of regions in a target frame which configures themoving image contents and another frame. In this manner, it is possibleto obtain a feature amount in each of the plurality of regions based onthe comparison result of the plurality of regions in the target framewhich configures the moving image contents and another frame.

According to the present application, it is possible to appropriatelyobtain imaging information at a time of an imaging operation. Inaddition, effects which are described here are not necessarily limited,and may be any one of effects which are described in the presentdisclosure.

Additional features and advantages are described herein, and will beapparent from the following Detailed Description and the figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram which illustrates a functional configurationexample of an image processing device according to a first embodiment ofthe present application;

FIG. 2 is a diagram which illustrates a configuration example of a CMOSimage sensor which is a base of the present application;

FIG. 3 is a diagram which schematically illustrates a relationshipbetween each line in the CMOS image sensor which is the base of thepresent application and an exposure period of each line;

FIGS. 4A to 4C are diagrams which illustrate images generated using animaging device including the CMOS image sensor which is the base of thepresent application, and a correction example thereof;

FIG. 5 is a diagram which illustrates an imaging direction of the CMOSimage sensor which is the base of the present application;

FIG. 6 is a diagram which illustrates a relationship among a motion ofthe CMOS image sensor which is the base of the present application, theimaging direction of the CMOS image sensor, and a generation form offocal plane distortion;

FIGS. 7A to 7E are diagrams which illustrate an example of a form of thefocal plane distortion which occurs in an image generated using the CMOSimage sensor which is the base of the present application;

FIGS. 8A to 8C are diagrams which schematically illustrate framesconfiguring moving image contents which become a determination targetusing a determination unit according to the first embodiment of thepresent application;

FIGS. 9A and 9B are diagrams which illustrate a relationship between aquantity of motion corresponding to a local movement vector which isdetected by the determination unit according to the first embodiment ofthe present application and a region in a frame;

FIG. 10 is a diagram which illustrates a relationship between thequantity of motion corresponding to the local movement vector which isdetected by the determination unit according to the first embodiment ofthe present application and the region in the frame;

FIG. 11 is a block diagram which illustrates a functional configurationexample of a correction unit according to the first embodiment of thepresent application;

FIGS. 12A to 12E are diagrams which illustrate a relationship between animage as a target of correction processing using the correction unitaccording to the first embodiment of the present application, an imageafter the correction and each piece of information used in thecorrection processing;

FIGS. 13A to 13D are diagrams which illustrate a relationship betweenthe image as the target of the correction processing using thecorrection unit according to the first embodiment of the presentapplication and each piece of information used in the correctionprocessing;

FIGS. 14A to 14C are diagrams which illustrate a relationship betweenthe image as the target of the correction processing using thecorrection unit according to the first embodiment of the presentapplication and each piece of information used in the correctionprocessing;

FIG. 15 is a flowchart which illustrates an example of processing orderof the correction processing in the image processing device according tothe first embodiment of the present application; and

FIG. 16 is a block diagram which illustrates a functional configurationexample of an image processing device according to a modificationexample of the embodiment of the present application.

DETAILED DESCRIPTION

Hereinafter, embodiments for executing the present application(hereinafter, referred to as embodiment) will be described. Descriptionswill be made in the following order.

1. First Embodiment (example in which imaging information at time ofimaging operation of moving image contents is determined using localmovement vector obtained from plurality of regions in frame)

2. Modification Example

1. First Embodiment Configuration Example of Image Processing Device

FIG. 1 is a block diagram which illustrates a functional configurationexample of an image processing device 100 according to a firstembodiment of the present application.

The image processing device 100 includes a moving image contentsrecording unit 110, a determination unit 120, a correction unit 130, andan output unit 140. The image processing device 100 is, for example, animage processing device such as an imaging device which generates movingimage contents by imaging an object, or a reproducing device whichreproduces moving image contents. The reproducing device is, forexample, a television, a digital video recorder, a hard disk drive (HDD)recorder, a personal computer, a game machine, a projector, and a mobileinformation processing device (for example, smart phone and tabletterminal). In addition, the imaging device is, for example, a digitalstill camera or a digital video camera (for example, camera-integratedrecorder) which can generate moving image contents by imaging an object.

The moving image contents recording unit 110 records moving imagecontents (moving image file), and supplies the recorded moving imagecontents to the determination unit 120 and the correction unit 130. Forexample, when the image processing device 100 is an imaging device,image data (moving image data) which is generated by an imaging unit(not shown) is recorded in the moving image contents recording unit 110as moving image contents (moving image file). In addition, the imagingunit is configured of, for example, a Charge Coupled Device (CCD) imagesensor, or a Complementary Metal Oxide Semiconductor (CMOS) imagesensor. In addition, when the image processing device 100 is areproducing device which does not have an imaging function, image data(moving image data) which is generated by another imaging device isrecorded in the moving image contents recording unit 110 as moving imagecontents (moving image file). In addition, similarly, when the imageprocessing device 100 is a conversion device (or editing device) whichdoes not have the imaging function, image data (moving image data) whichis generated by another imaging device is recorded in the moving imagecontents recording unit 110 as moving image contents (moving imagefile). In addition, the conversion device (or editing device) is, forexample, a device which performs correction processing such as acorrection of focal plane distortion with respect to moving imagecontents of which imaging information at a time of imaging operation isnot known using software for editing moving image contents, software forreproducing moving image contents, or the like. In addition, a devicewhich performs correction processing such as a correction of focal planedistortion with respect to moving image contents of which imaginginformation at a time of imaging operation is not known may be used as aconversion device (or editing device), in a cloud environment, or thelike.

The determination unit 120 determines imaging information at a time ofthe imaging operation with respect to moving image contents which arerecorded in the moving image contents recording unit 110, and outputs adetermination result thereof (imaging information) to the correctionunit 130. Here, the imaging information at the time of the imagingoperation of moving image contents is a scheme of capturing pixel valuesof the image sensor (imaging element) at the time of imaging operationof the moving image contents. The capturing scheme is at least one ofthe imaging direction of the imaging sensor at the time of imagingoperation and a shutter system of the image sensor.

For example, the determination unit 120 obtains a feature amount (forexample, local movement vector) from a plurality of regions in a framewhich configures moving image contents which are recorded in the movingimage contents recording unit 110. In addition, the determination unit120 determines a capturing scheme thereof using the obtained featureamount. In addition, the plurality of regions in the frame whichconfigures moving image contents, and a local movement vector which isobtained from the regions will be described in detail with reference toFIGS. 8A to 8C. In addition, a determining method using thedetermination unit 120 will be described in detail with reference toFIGS. 8A to 10.

The correction unit 130 performs a correction of focal plane distortionwith respect to moving image contents which are recorded in the movingimage contents recording unit 110 based on a determination result(imaging information) which is output from the determination unit 120.In addition, the correction unit 130 outputs the image (corrected image)which is subjected to the correction processing to the output unit 140.

The output unit 140 outputs the moving image contents which aresubjected to the correction processing using the correction unit 130.The output unit 140 can be set to a display unit which displays movingimage contents, or an output unit which outputs the moving imagecontents to another device, for example.

Regarding Principle of Occurring Focal Plane Distortion

Here, an operation of a focal plane shutter of a CMOS image sensor and aprinciple of occurring focal plane distortion will be described. TheCMOS image sensor is an image sensor which is generally used as an imagesensor (imaging element) of an imaging device (for example, digitalvideo camera (for example, camera-integrated recorder)).

Configuration Example of CMOS Image Sensor

FIG. 2 is a diagram which illustrates a configuration example of a CMOSimage sensor 200 which is a base of the present application. Inaddition, the CMOS image sensor 200 is an example of a general CMOSimage sensor.

The CMOS image sensor 200 includes pixels 211 to 219, analog signallines (vertical signal lines) 221 to 223, A/D converters 231 to 233, anda digital signal line 240. In addition, in FIG. 2, the pixels 211 to 219which are configured of a photodiode, an amplifier, and the like, areschematically illustrated using rectangles. In addition, in FIG. 2, forease of description, only nine pixels 211 to 219 are illustrated, andother pixels are not shown. In addition, only analog signal lines 221 to223 and A/D converters 231 to 233 which correspond to the pixels 211 to219 are illustrated, and others are not shown.

The pixels 211 to 219 are arranged in a lattice shape in the pixelregion 201 of the CMOS image sensor 200. In addition, the pixels 211 to213 are connected to the A/D converter 231 through the analog signalline 221 in the vertical direction. Similarly, the pixels 214 to 216 areconnected to the A/D converter 232 through the analog signal line 222 inthe vertical direction, and pixels 217 to 219 are connected to the A/Dconverter 233 through the analog signal line 223 in the verticaldirection.

In addition, in the pixels 211 to 219, the photodiode (light receivingunit) receives light, and the light is accumulated by being convertedinto a charge. In addition, the accumulated charge is converted into avoltage, and is amplified by the amplifier. The voltage which isamplified in this manner is transferred to the analog signal lines 221to 223 in each line (each row) due to ON/OFF of a pixel selectionswitch.

The A/D converters 231 to 233 are an A/D converter which turns an analogsignal of each pixel which is input through the analog signal lines 221to 223 into a digital signal by converting the analog signal, andoutputs the converted digital signal to the digital signal line 240.

In this manner, the CMOS image sensor 200 includes the A/D converters231 to 233 of the same number as the number of pixels in the horizontaldirection. For this reason, it is possible to use the A/D converters 231to 233 by sharing the converters with pixels in the vertical direction.In this manner, it is possible to reduce a circuit area andmanufacturing cost. It is possible to simultaneously perform the A/Dconversion with respect to the pixels in the horizontal direction,however, it is not possible to simultaneously perform the A/D conversionwith respect to the pixels in the vertical direction. For this reason,it is necessary to delay a start timing of an exposure period by aperiod of time in which at least the A/D conversion is performed withrespect to the pixels in the vertical direction. The exposure periodwill be described in detail with reference to FIG. 3.

Example of Exposure Period of CMOS Image Sensor

FIG. 3 is a diagram which schematically illustrates a relationshipbetween each line in the CMOS image sensor 200 which is a base of thepresent application and an exposure period of each the line. In FIG. 3,a graph is illustrated in which a horizontal axis is set to a time axis,and a vertical axis is set to an axis which denotes the exposure periodwith respect to the pixel (light receiving unit) in each line. Forexample, a line 1 which is attached to the vertical axis is set tocorrespond to a line including pixels 211, 214, and 217 (a line inhorizontal direction) which are illustrated in FIG. 2. Similarly, a line2 which is attached to the vertical axis is set to correspond to a lineincluding pixels 212, 215, and 218 which are illustrated in FIG. 2, anda line N which is attached to the vertical axis is set to correspond toa line including pixels 213, 216, and 219 which are illustrated in FIG.2.

In addition, in FIG. 3, an example of an exposure period in a case inwhich a shutter is clicked using an electronic shutter system as anexposure method (hereinafter, referred to as focal plane shutter system)is illustrated. In addition, the focal plane shutter system is alsoreferred to as a rolling shutter system.

As described above, since the A/D converter is shared with pixels in thevertical direction, it is necessary to delay a processing timing of anA/D conversion period (period which is necessary when A/D converterperforms A/D conversion) CP at least as much as the processing timing ineach line which is illustrated in FIG. 3. In addition, since an exposureperiod EP is constant in each line (lines 1 to N) in a general imagingmode, a temporal difference dt (A/D conversion period CP) occurs in theexposure timing in each line (lines 1 to N) in the vertical direction.

Accordingly, as illustrated in FIG. 3, it is necessary to delay exposuretiming by dt between an exposure start timing T1 in line 1 and anexposure start timing T2 in line 2.

In addition, similarly, it is necessary to delay exposure timing by dtbetween an exposure start timing in one upper line and an exposure starttiming in the subsequent line with respect to each of the subsequentlines 2 to N.

In this manner, since the focal plane shutter system is a system inwhich a shutter is sequentially clicked in each line, exposure timing inthe upper region and the lower region in one frame is different in eachline. For this reason, there is a concern that distortion may occur in amoving object. For example, when a posture of an imaging device whichincludes the CMOS image sensor 200 is changed during an exposure period,or when an object moves, distortion occurs in an image which is outputfrom the CMOS image sensor 200. Examples of distortion of the outputimage are illustrated in FIGS. 4A to 4C.

Example of Focal Plane Distortion

FIGS. 4A to 4C are diagrams which illustrate an image which is generatedusing an imaging device which includes the CMOS image sensor 200 whichis the base of the present application, and a correction examplethereof. In addition, in FIGS. 4A and 4B, the same object is set to animaging target, and an example of an image in a case in which a state ofthe imaging device is changed is illustrated.

FIG. 4A illustrates an image 250 in a case in which the image isphotographed in a state of stopping the imaging device.

FIG. 4B illustrates an image 251 in a case in which the image isphotographed while moving the imaging device in the horizontaldirection. In this manner, distortion (referred to as focal planedistortion) due to the focal plane shutter system occurs in the image251 which is photographed while moving the imaging device in thehorizontal direction.

Here, it is possible to generate an image with no focal plane distortionby correcting the focal plane distortion which has occurred in thismanner. The correction is referred to as a correction of focal planedistortion. For example, it is possible to generate an image with nofocal plane distortion (that is, approximately the same image as image250) from the image 251 with the focal plane distortion.

However, since there are a plurality of imaging directions in the CMOSimage sensor 200, it is necessary to correct focal plane distortion bytaking an imaging direction into consideration. The imaging direction isillustrated in FIG. 5. In addition, FIG. 4C illustrates an example of animage (image 252) which is corrected based on a wrong imaging direction.

Example of Imaging Direction of CMOS Image Sensor

FIG. 5 is a diagram which illustrates an imaging direction of the CMOSimage sensor 200 which is the base of the present application. FIG. 5illustrates four imaging directions of 1 to 4 as the imaging directionof the CMOS image sensor 200. In addition, in FIG. 5, each pixel isdenoted using a rectangle to which a mark (Red (R), Green (G), and Blue(B)) denoting a corresponding color is attached. In addition, in a pixelregion 201 of the CMOS image sensor 200, only a part of pixels isdenoted, and other pixels are not shown.

The imaging direction 1 is an imaging direction in which the upper leftcorner of the CMOS image sensor 200 which is illustrated in FIG. 5 isset to a start position of reading, and after the start position ofreading, as denoted by a dotted arrow 261, reading of each line issequentially performed in each one row so that the reading proceeds fromthe upper side to the lower side.

The imaging direction 2 is an imaging direction in which the upper rightcorner of the CMOS image sensor 200 which is illustrated in FIG. 5 isset to a start position of reading, and after the start position ofreading, as denoted by a dotted arrow 262, reading of each line issequentially performed in each one row so that the reading proceeds fromthe upper side to the lower side.

The imaging direction 3 is an imaging direction in which the lower leftcorner of the CMOS image sensor 200 which is illustrated in FIG. 5 isset to a start position of reading, and after the start position ofreading, as denoted by a dotted arrow 263, reading of each line issequentially performed in each one row so that the reading proceeds fromthe lower side to the upper side.

The imaging direction 4 is an imaging direction in which the lower rightcorner of the CMOS image sensor 200 which is illustrated in FIG. 5 isset to a start position of reading, and after the start position ofreading, as denoted by a dotted arrow 264, reading of each line issequentially performed in each one row so that the reading proceeds fromthe lower side to the upper side.

Here, it is necessary for a general image sensor (for example, CMOSimage sensor or CCD image sensor) to flexibly correspond to an attachingposition of a fixed camera, or to correspond to a posture of a handcamera. For this reason, a general image sensor is designed so that areading direction of a pixel in the horizontal direction (hereinafter,referred to as horizontal imaging direction), and a reading direction ofa pixel in the vertical direction (hereinafter, referred to as verticalimaging direction) can be changed. That is, an imaging direction of ageneral image sensor becomes the four imaging directions of 1 to 4 whichare illustrated in FIG. 5.

Here, in a general CMOS image sensor, the vertical imaging direction inthe imaging directions influences an occurrence form of focal planedistortion. As described above, since it is possible to perform imagingsimultaneously using pixels in the horizontal direction, the occurrenceform of focal plane distortion is not influenced when being photographedin any direction. The occurrence form of focal plane distortion will bedescribed in detail with reference to FIGS. 6 to 7E.

Occurrence Form of Focal Plane Distortion

FIG. 6 is a diagram which illustrates a relationship among a motion ofthe CMOS image sensor 200 which is the base of the present application,an imaging direction of the CMOS image sensor 200, and an occurrenceform of focal plane distortion.

FIGS. 7A to 7E are diagrams which illustrate an example of a form offocal plane distortion which occurs in an image which is generated bythe CMOS image sensor 200 which is the base of the present application.In addition, FIGS. 7A to 7E correspond to marks of an imaged image 273which is illustrated in FIG. 6.

Here, when a vertical imaging direction 272 illustrated in FIG. 6 is a“forward direction”, it means that the vertical imaging direction in theCMOS image sensor 200 is a direction from the top to the bottom (forexample, cases of imaging directions 1 and 2 illustrated in FIG. 5). Inaddition, when the vertical imaging direction 272 illustrated in FIG. 6is a “backward direction”, it means that the vertical imaging directionin the CMOS image sensor 200 is a direction from the bottom to the top(for example, cases of imaging directions 3 and 4 illustrated in FIG.5).

As illustrated in FIGS. 6 to 7E, the occurrence form of focal planedistortion is different according to a combination of the motion and theimaging direction of the CMOS image sensor 200.

Specifically, FIG. 7A illustrates an image example when the CMOS imagesensor 200 does not move (that is, a case in which imaging operation isperformed in a state of stopped imaging device). That is, when there isno motion of the CMOS image sensor 200, focal plane distortion does notoccur whatever the imaging direction of the CMOS image sensor 200 is.

In addition, FIGS. 7B and 7C illustrate an image example when there is amotion of the CMOS image sensor 200 in the horizontal direction (thatis, a case in which imaging operation is performed by panning imagingdevice). As illustrated in FIGS. 7B and 7C, when there is a motion ofthe CMOS image sensor 200 in the horizontal direction, focal planedistortion occurs since an imaging position in each line is deviated inthe horizontal direction.

Here, when there is a motion of the CMOS image sensor 200 in thehorizontal direction, focal plane distortion becomes different accordingto a direction of the motion of the CMOS image sensor 200. Specifically,a case in which the motion of the CMOS image sensor 200 in thehorizontal direction is a motion from the left to the right in thehorizontal direction is assumed. In this manner, when the motion of theCMOS image sensor 200 in the horizontal direction is the motion from theleft to the right in the horizontal direction, and the vertical imagingdirection of the CMOS image sensor 200 is the forward direction, focalplane distortion illustrated in FIG. 7B occurs. On the other hand, whenthe motion of the CMOS image sensor 200 in the horizontal direction isthe motion from the left to the right in the horizontal direction, andthe vertical imaging direction of the CMOS image sensor 200 is thebackward direction, focal plane distortion illustrated in FIG. 7Coccurs.

In addition, a case in which a motion of the CMOS image sensor 200 inthe horizontal direction is a motion from the right to the left in thehorizontal direction is assumed. In this manner, when the motion of theCMOS image sensor 200 in the horizontal direction is the motion from theright to the left in the horizontal direction, and the vertical imagingdirection of the CMOS image sensor 200 is the forward direction, focalplane distortion illustrated in FIG. 7C occurs. On the other hand, whenthe motion of the CMOS image sensor 200 in the horizontal direction isthe motion from the right to the left in the horizontal direction, andthe vertical imaging direction of the CMOS image sensor 200 is thebackward direction, focal plane distortion illustrated in FIG. 7Boccurs.

In this manner, the relationship between the motion of the CMOS imagesensor 200 and focal plane distortion is reversed depending on whetherthe vertical imaging direction is the forward direction or the backwarddirection.

In addition, FIGS. 7D and 7E illustrate image examples in a case inwhich there is a motion of the CMOS image sensor 200 in the verticaldirection (that is, a case in which imaging operation is performed bytilting imaging device). As illustrated in FIGS. 7D and 7E, when thereis a motion of the CMOS image sensor 200 in the vertical direction,focal plane distortion occurs due to deviation of an imaging position ineach line in the vertical direction.

In addition, similarly to the case in which there is the motion of theCMOS image sensor 200 in the vertical direction, a relationship betweenthe motion of the CMOS image sensor 200 and focal plane distortion isreversed depending on whether the vertical imaging direction is theforward direction or the backward direction, as illustrated in FIG. 6.

Here, when moving image contents which are generated by the CMOS imagesensor are recorded in the imaging device, there is no case in which thevertical imaging direction of the CMOS image sensor at the time of theimaging operation is recorded by being related to the moving imagecontents as meta information. For this reason, when the moving imagecontents are reproduced in another reproducing device, the reproducingdevice is incapable of ascertaining the vertical imaging direction ofthe CMOS image sensor at the time of imaging operation of the movingimage contents which are reproducing targets.

In this manner, when a correction of focal plane distortion is performedwith respect to moving image contents of which the vertical imagingdirection of the CMOS image sensor 200 at the time of imaging operationis not known, if correction processing is performed by taking a wrongvertical imaging direction, it causes a reverse correction, anddistortion increases.

For example, a case in which a correction of focal plane distortion isperformed with respect to the image 251 (image including focal planedistortion) illustrated in FIG. 4B is assumed. In this case, when thecorrection of focal plane distortion is performed by taking a wrongvertical imaging direction, a distortion level further increases likethe image 252 illustrated in FIG. 4C, and there is a concern that thecorrection of focal plane distortion may fail.

Therefore, according to the embodiment of the present application, acase is exemplified in which imaging information at the time of imagingoperation is appropriately obtained with respect to moving imagecontents of which imaging information at the time of imaging operationis not known. In addition, a correction of focal plane distortion isperformed in an appropriate direction using the obtained imaginginformation (for example, vertical imaging direction or shutter system).In this manner, it is possible to appropriately perform the correctionof focal plane distortion.

Determination Example of Imaging Information

FIGS. 8A to 8C are diagrams which schematically illustrate frames whichconfigure moving image contents 300 which are determination targets ofthe determination unit 120 according to the first embodiment of thepresent application.

FIG. 8A schematically illustrates each frame which configures the movingimage contents 300 which are recorded in the moving image contentsrecording unit 110 in time sequence. Here, in FIG. 8A, each frame isschematically illustrated as an outlined rectangle, and is illustratedby being attached with a mark (serial number) for identifying the framein each rectangle. For example, it is set such that a frame 1 is a topframe in each frame which configures the moving image contents 300, andframes 2 to N are set to subsequent frames from the frame 1.

In FIGS. 8B and 8C, two continuous frames among frames which configurethe moving image contents 300 are illustrated. Specifically, in FIG. 8B,a frame n−1 (310) which configures the moving image contents 300 isillustrated, and in FIG. 8C, a frame n (320) which configures the movingimage contents 300 is illustrated. In addition, the frame n−1 (310)illustrated in FIG. 8B corresponds to frames 1 to N−1 which configurethe moving image contents 300. In addition, the frame n (320)illustrated in FIG. 8C corresponds to frames 2 to N which configure themoving image contents 300.

In addition, in FIGS. 8B and 8C, an example in which each frame (framen−1 (310), frame n (320)) is divided into eight regions (0th region toseventh region), and a movement vector is detected from each of theregions (0th region to seventh region) is illustrated. In addition, inFIGS. 8B and 8C, an example of dividing into eight regions (0th regionto seventh region) in the vertical direction as a dividing method isillustrated.

Here, a movement vector which is detected from a plurality of regions(0th region to seventh region) in each frame is a local movement vectorin each frame, and is also referred to as a Local Motion vector (LMV).For this reason, hereinafter, movement vectors which are detected fromthe plurality of regions (0th region to seventh region) in each framewill be described as a local movement vector or a LMV.

In addition, as a method of detecting a local movement vector, forexample, it is possible to use a method of obtaining a local movementvector by retrieving a position with strong correlation from aretrieving region of the previous frame with respect to a target regionwhich becomes a detection target of the local movement vector. As thedetection method, for example, it is possible to use a block matchingmethod (for example, refer to Japanese Unexamined Patent ApplicationPublication No. 2007-208580). The block matching method is a methodwhich retrieves a location of an image similar to an image included in atarget region which becomes a detection target of a local movementvector in the previous frame which is a comparison target, and detectsthe local movement vector of the target region based on a retrievingresult thereof. Specifically, the local movement vector is detected bysetting a searching range to a magnitude of the greatest quantity ofmotion which is assumed with respect to the plurality of regions(blocks) in the target region, and performing retrieval in the setsearch range.

In addition, another detection method may be used. For example, it ispossible to use a detection method in which the local movement vector isdetected using a numeric calculation, using a method such as an opticalflow.

In addition, in the frame n−1 (310), regions on the upper side (0thregion to third region) are denoted by being surrounded with arectangular dotted line as the upper region 311, and regions on thelower side (fourth region to seventh region) are denoted by beingsurrounded with a rectangular dotted line as the lower region 312.Similarly, in the frame n (320), regions on the upper side (0th regionto third region) are denoted by being surrounded with a rectangulardotted line as the upper region 321, and regions on the lower side(fourth region to seventh region) are denoted by being surrounded with arectangular dotted line as the lower region 322. In addition, the upperregion 311, the lower region 312, the upper region 321, and the lowerregion 322 will be described in detail with reference to FIGS. 9A and9B.

In this manner, the determination unit 120 obtains a feature amount(local movement vector) in each of the plurality of regions based on acomparison result between a plurality of regions in a target frame(current frame) which configures moving image contents and another frame(previous frame to current frame).

In addition, according to the first embodiment of the presentapplication, an example in which one frame is divided into eightregions, and local movement vectors are detected has been described,however, movement vectors which are detected with respect to regionsdivided using another dividing method may be used.

FIGS. 9A and 9B are diagrams which illustrate a relationship between aquantity of motion corresponding to a local movement vector which isdetected by the determination unit 120 according to the first embodimentof the present application and a region in a frame. In addition, inFIGS. 9A and 9B, an example of a case in which a moving body is notincluded in a frame is illustrated.

In FIGS. 9A and 9B, respective regions (eight regions for detectinglocal movement vector) of each frame (for example, frames n−1 to n+2)are denoted side by side in a frame unit on a horizontal axis. Inaddition, a value (quantity of motion) corresponding to the localmovement vector which is detected in each region is denoted on avertical axis. In addition, it is set such that the frames n−1 to n+2correspond to the frames n−1 to n+2 which are illustrated in FIG. 8A.

For example, a case in which an imaging operation is performed using animaging device which generates moving image contents which are recordedin the moving image contents recording unit 110 is assumed. Here, it isset such that the imaging operation is performed while moving theimaging device in the horizontal direction. In this case, since exposuretiming in the vertical direction becomes different when a moving speedin the horizontal direction is different, it is assumed that a movementvector of each region in the same frame becomes different.

For example, FIG. 9A illustrates a relationship between a region in eachframe and a quantity of motion corresponding to a local movement vectordetected from each region when the imaging operation is performed bymoving the imaging device in the horizontal direction while changing amoving speed.

As described above, since exposure timing in the vertical direction isdeviated, a quantity of motion corresponding to a movement vector whichis detected from each region is changed according to the change in themoving speed. In addition, a change in the quantity of motioncorresponding to the movement vector which is detected from a regioncorresponding to a boundary between frames becomes different dependingon a vertical imaging direction in the CMOS image sensor.

For example, a case in which the vertical imaging direction in the CMOSimage sensor is the forward direction is assumed. In this case, when aquantity of motion (speed, difference from the previous frame)corresponding to a local movement vector which is detected from eachregion is arranged in each region, a change in the quantity of motionbecomes smooth, as illustrated in FIG. 9A. In addition, a change in thequantity of motion at a boundary of each frame also becomes smooth. Thatis, it is possible to denote the change in the quantity of motion usinga curve 330.

On the other hand, a case in which the vertical imaging direction in theCMOS image sensor is the backward direction is assumed. In this case,when the quantity of motion (speed, difference from the previous frame)corresponding to a local movement vector which is detected from eachregion is arranged in each region, the change in the quantity of motionin each frame becomes smooth as illustrated in FIG. 9B, however, thequantity of motion at the boundary of each frame rapidly changes. Inaddition, as illustrated in FIG. 9B, the change in a quantity of motionbetween each of frames becomes opposite to the example illustrated inFIG. 9A.

In this manner, the change in the quantity of motion in each framebecomes opposite in a case in which the vertical imaging direction isthe forward direction, and in a case in which the vertical imagingdirection is the backward direction, and the change in the quantity ofmotion at the boundary of each frame becomes different. Therefore,according to the first embodiment of the present application, an exampleis described in which whether the vertical imaging direction is theforward direction or the backward direction is determined by making useof these properties.

Calculation Example of Local Movement Vector

First, the determination unit 120 sets one frame among frames whichconfigure the moving image contents 300 to the current frame (targetframe). Subsequently, the determination unit 120 divides the currentframe into eight regions, and detects a local movement vector in each ofthe regions. For example, a case in which nth frame n (frame n (320)illustrated in FIG. 8C) among frames which configure the moving imagecontents 300 is set to the current frame (target frame) is assumed. Inthis case, the determination unit 120 detects local movement vectorswith respect to regions 0 to 7 in the frame n (320).

Calculation Example of Quantity of Motion of Forward Direction FrameBoundary and Quantity of Motion of Backward Direction Frame Boundary

Subsequently, the determination unit 120 calculates a quantity of motionof a forward direction frame boundary and a quantity of motion of abackward direction frame boundary with respect to the frame n (320). Forexample, the determination unit 120 calculates the quantity of motion ofthe forward direction frame boundary (NormalFrameDiff_(n)) using thefollowing formula 1, and calculates the quantity of motion of thebackward direction frame boundary (InverseFrameDiff_(n)) using thefollowing formula 2. In addition, the suffix n is a serial number foridentifying each frame.

$\begin{matrix}{{NormalFrameDiff}_{n} = {{{\sum\limits_{i = 0}^{3}\;{{LMV}\lbrack i\rbrack}_{n}} - {\sum\limits_{i = 4}^{7}\;{{LMV}\lbrack i\rbrack}_{n - 1}}}}} & (1) \\{{InverseFrameDiff}_{n} = {{{\sum\limits_{i = 4}^{7}\;{{LMV}\lbrack i\rbrack}_{n}} - {\sum\limits_{i = 0}^{3}\;{{LMV}\lbrack i\rbrack}_{n - 1}}}}} & (2)\end{matrix}$

Here, the first section of the formula 1 denotes a total sum of aquantity of motion of LMVs of the upper region 321 in an image of thecurrent frame (target frame). That is, the first section denotes a totalsum of a quantity of motion of the LMVs of respective regions of 0 to 3(region surrounded with rectangle (upper region 321)) which areillustrated in FIG. 8C. In addition, the second section of the formula 1denotes a total sum of a quantity of motion of LMVs in the lower region312 in an image in the previous frame (frame n−1) to the current frame.That is, the second section denotes a total sum of a quantity of motionof LMVs of respective regions of 4 to 7 (region surrounded withrectangle (lower region 312)) which are illustrated in FIG. 8B.

In addition, the quantity of motion of the forward direction frameboundary (NormalFrameDiff_(n)) which is calculated using the formula 1denotes an amount of change of a movement vector at a frame boundaryportion when the vertical imaging direction is the forward direction.

In addition, the first section of the formula 2 denotes a total sum of amovement vector of LMVs of the lower region 322 in an image of thecurrent frame (target frame). That is, the first section denotes a totalsum of a quantity of motion of the LMVs of respective regions of 4 to 7(region surrounded with rectangle (lower region 322)) which areillustrated in FIG. 8C. In addition, the second section of the formula 2denotes a total sum of a quantity of motion of LMVs in the upper region311 in an image in the previous frame (frame n−1) to the current frame.That is, the second section denotes a total sum of a quantity of motionof LMVs of respective regions of 0 to 3 (region surrounded withrectangle (upper region 311)) which are illustrated in FIG. 8B.

In addition, the quantity of motion of the backward direction frameboundary (InverseFrameDiff_(n)) which is calculated using the formula 2denotes an amount of change of a movement vector at a frame boundaryportion when the vertical imaging direction is the backward direction.

Subsequently, the determination unit 120 similarly calculates thequantity of motion of the forward direction frame boundary(NormalFrameDiff_(n)) and the quantity of motion of the backwarddirection frame boundary (InverseFrameDiff_(n)) with respect to eachframe after the n+1th frame n+1.

In addition, in the example, an example is described in which thequantity of motion of a forward direction frame boundary and thequantity of motion of a backward direction frame boundary are calculatedusing all of the LMVs in each region of the upper region, and all of theLMVs in each region of the lower region in the image in each frame.However, the quantity of motion of the forward direction frame boundaryand the quantity of motion of the backward direction frame boundary maybe calculated using only a part of the LMVs in each region of the upperregion and the LMVs in each region of the lower region in the image ineach frame. For example, the quantity of motion of the forward directionframe boundary, and the quantity of motion of the backward directionframe boundary may be calculated using the LMVs of a part of the regionsin the upper part (for example, regions 0 and 1), and the LMVs of a partof the regions in the lower part (for example, regions 6 and 7) in theimage in each frame. In addition, the quantity of motion of the forwarddirection frame boundary, and the quantity of motion of the backwarddirection frame boundary may be calculated using the LMV of one regionin the upper part (for example, region 0), and the LMV of one region inthe lower part (for example, region 7) in the image in each frame.

In addition, hitherto, an example in which movement vectors which aredetected from eight regions (0th region to seventh region) in thevertical direction (y direction) of a frame are used has been described,for ease of description. However, in the above described block matchingmethod, in general, a frame is divided into a matrix, and a localmovement vector is obtained from a two-dimensional division region. Forthis reason, the two-dimensional movement vector may be used. Forexample, a case is assumed in which one frame is divided into 64 regionsto have 8 regions in the horizontal direction (x direction) and 8regions in the vertical direction (y direction), and movement vectorswhich are detected from each of the regions are used. In this case, forexample, a total sum of moving quantities of LMVs of 32 regions (8(xdirection)×4(y direction)) corresponding to the upper region in theimage of the current frame (frame n) is obtained in the first section ofthe formula 1. In addition, similarly, a total sum of moving quantitiesof LMVs of 32 (8(x direction)×4(y direction)) regions is obtained withrespect to each of other sections (second section of formula 1, firstand second sections of formula 2). In addition, the quantity of motionof the forward direction frame boundary and quantity of motion of thebackward direction frame boundary are calculated using each value ofthese.

In addition, the first section of the formula 1 may be calculated bybeing limited to an effective LMV. That is, a mean value of LMVs may beobtained by being divided by the number of LMVs (number of effectiveLMVs) which are a calculation target, and the mean value of the LMVs maybe used instead of the value of the first section of the formula 1. Forexample, when one frame is divided into 64 (8(x direction)×8(ydirection)) regions, a mean value of LMVs is obtained by dividing avalue which is obtained by the first section of formula 1 by the numberof effective LMVs (maximum 32 (8(number of LMVs in x direction)×4(numberof LMVs in y direction))). In addition, the mean value of the LMVs isused instead of the value in the first section of formula 1. Here, anineffective LMV is subject to a value of 0. In addition, when noeffective LMVs are present in the target region, the value of the firstsection of formula 1 is set to 0. In addition, similarly, a value ofeach section in other sections (second section in formula 1, first andsecond sections in formula 2) is also calculated by being limited toeffective LMVs. In this manner, when calculating the quantity of motionof the forward direction frame boundary and quantity of motion of thebackward direction frame boundary, the quantities can be calculated bybeing limited to effective LMVs.

In this manner, it is possible to improve determination precision ofimaging information by calculating the quantity of motion of the forwarddirection frame boundary and the quantity of motion of the backwarddirection frame boundary, by being limited to effective LMVs.

In addition, in the example, a case in which eight movement vectors areobtained and used in one frame period has been exemplified, however, asdescribed above, two or more movement vectors (for example, movementvectors of 2 to 7, or 9 or more) may be obtained and used in one frameperiod.

Calculation Example of Integration Value of Quantity of Motion of theForward Direction Frame Boundary, and Integration Value of Quantity ofMotion of the Backward Direction Frame Boundary

Here, a case is assumed in which hand shake occurs in an imagingoperation using a general CMOS image sensor. For example, when avertical imaging direction of the CMOS image sensor in the imagingoperation is the forward direction, an LMV which becomes a curve 330illustrated in FIG. 9A (LMV Index=0 to 7) is obtained in each frame. Onthe other hand, when the vertical imaging direction of the CMOS imagesensor in the imaging operation is the backward direction, an LMVillustrated in FIG. 9B (LMV Index=0 to 7) is obtained in each frame.

Therefore, an amount of change of a movement vector at a frame boundaryportion is calculated in order to determine a case in which the verticalimaging direction is the forward direction, and a case in which thevertical imaging direction is the backward direction.

That is, the determination unit 120 integrates (adds) the respectivecalculated quantity of motion of the forward direction frame boundary(NormalFrameDiff_(n)) and quantity of motion of the backward directionframe boundary (InverseFrameDiff_(n)). Specifically, the determinationunit 120 calculates an integration value of quantity of motion of theforward direction frame boundary (NormalDiffSum) by integrating (adding)the quantity of motion of the forward direction frame boundary(NormalFrameDiff_(n)) using the following formula 3. In addition, thedetermination unit 120 calculates an integration value of quantity ofmotion of the backward direction frame boundary (InverseDiffSum) byintegrating (adding) the quantity of motion of the backward directionframe boundary (InverseFrameDiff_(n)) using the following formula 4.

$\begin{matrix}{{NormalDiffSum} = {\sum\limits_{n = 1}^{N}\;{NormalFrameDiff}_{n}}} & (3) \\{{InverseDiffSum} = {\sum\limits_{n = 1}^{N}\;{InverseFrameDiff}_{n}}} & (4)\end{matrix}$

Here, N is a value denoting the number of frames (integer of 1 or more)which configure moving image contents.

Another Calculation Example of Integration Value of Quantity of Motionof the Forward Direction Frame Boundary, and Integration Value ofQuantity of Motion of the Backward Direction Frame Boundary

Formulas 3 and 4 are formulas which are used when calculating anintegration value of all of frames (first frame to last frame) whichconfigure moving image contents. However, since there is no previousframe to the first frame among frames which configure the moving imagecontents, it is also assumed that a local movement vector is notdetected. In this manner, when a local movement vector is not detectedin the first frame, it is not possible to perform a calculation of thesecond section in formula 1 and formula 2, respectively. Therefore, theintegration value of quantity of motion of the forward direction frameboundary (NormalDiffSum) may be calculated using the following formula5, and the integration value of quantity of motion of the backwarddirection frame boundary (InverseDiffSum) may be calculated using thefollowing formula 6. However, in formulas 5 and 6, N is set to aninteger of 2 or more.

$\begin{matrix}{{NormalDiffSum} = {\sum\limits_{n = 2}^{N}\;{NormalFrameDiff}_{n}}} & (5) \\{{InverseDiffSum} = {\sum\limits_{n = 2}^{N}\;{InverseFrameDiff}_{n}}} & (6)\end{matrix}$

Determination Example of Vertical Imaging Direction

Subsequently, the determination unit 120 determines the vertical imagingdirection by comparing the calculated integration value of quantity ofmotion of the forward direction frame boundary (NormalDiffSum) and anintegration value of quantity of motion of the backward direction frameboundary (InverseDiffSum) with each other.

As described above, a direction in which an amount of change of a localmovement vector of the frame boundary portion is smoothly connected isconsidered to be an actual imaging direction. Therefore, it is possibleto determine that the direction of being smoothly connected is theactual imaging direction by comparing the integration value of quantityof motion of the forward direction frame boundary (NormalDiffSum) andthe integration value of quantity of motion of the backward directionframe boundary (InverseDiffSum) with each other.

Specifically, the determination unit 120 determines whether or not theintegration value of quantity of motion of the forward direction frameboundary (NormalDiffSum) is smaller than the integration value ofquantity of motion of the backward direction frame boundary(InverseDiffSum). In addition, the determination unit 120 determinesthat the vertical imaging direction is the forward direction when theintegration value of quantity of motion of the forward direction frameboundary (NormalDiffSum) is smaller than the integration value ofquantity of motion of the backward direction frame boundary(InverseDiffSum). On the other hand, the determination unit 120determines that the vertical imaging direction is the backward directionwhen the integration value of quantity of motion of the forwarddirection frame boundary (NormalDiffSum) is equal to or greater than theintegration value of quantity of motion of the backward direction frameboundary (InverseDiffSum).

That is, the determination unit 120 makes the determination using thefollowing determination condition (formula 7), and can obtain thevertical imaging direction as the imaging information.

$\begin{matrix}{{{Imaging}\mspace{14mu}{Information}} = \left\{ \begin{matrix}{{Vertical}\mspace{14mu}{imaging}\mspace{14mu}{forward}\mspace{14mu}{directon}} & \begin{matrix}{{{Normal}\mspace{14mu}{Diff}\mspace{14mu}{Sum}} <} \\{{Inverse}\mspace{14mu}{Diff}\mspace{14mu}{Sum}}\end{matrix} \\{{Vertical}\mspace{14mu}{imaging}\mspace{14mu}{backward}\mspace{14mu}{direction}} & {Otherwise}\end{matrix} \right.} & (7)\end{matrix}$

In this manner, it is possible to determine the vertical imagingdirection by integrating amounts of change of movement vectors at theframe boundary portion when the vertical imaging direction is theforward direction, and when the vertical imaging direction is thebackward direction, and by comparing the amounts of change with eachother.

Determination Example of Moving Image Contents of Global Shutter System

It is possible to determine the vertical imaging direction of an imagesensor at the time of imaging operation with respect to moving imagecontents of which occurrence of focal plane distortion is known usingthe above described determination method. In addition, it is possible toappropriately perform a correction of focal plane distortion based onthe determination result (vertical imaging direction).

However, for example, moving image contents which are imaged using theimage sensor (for example, CCD image sensor) of the global shuttersystem do not have focal plane distortion in principle. For this reason,it is not necessary to perform a correction of focal plane distortionwith respect to moving image contents which are imaged using a globalshutter image sensor.

In this manner, there is a case in which a correction of focal planedistortion is necessary, and a case in which the correction of focalplane distortion is not necessary according to a shutter system of animage sensor. Therefore, in order to correspond to moving image contentswhich are generated by the global shutter image sensor (for example, CCDimage sensor), it is preferable to determine whether the shutter systemis the global shutter system or the focal plane shutter system.

Therefore, in a case of not knowing the imaging information at the timeof imaging operation of moving image contents, a determination of theshutter system (global shutter system, focal plane shutter system) isperformed along with a determination of the vertical imaging direction(forward direction, backward direction). In addition, the correctionprocessing is performed based on the determination result.

FIG. 10 is a diagram which illustrates a relationship between a quantityof motion corresponding to a local movement vector which is detected bythe determination unit 120 according to the first embodiment of thepresent application and a region in a frame. In addition, a graphillustrated in FIG. 10 corresponds to the graphs illustrated in FIGS. 9Aand 9B. In addition, FIG. 10 illustrates an example of a relationshipbetween a quantity of motion corresponding to a local movement vectorwhich is detected with respect to the moving image contents generated bythe global shutter image sensor (for example, CCD image sensor) and aregion in a frame. In addition, FIG. 10 illustrates an example of a casein which there is no moving body in the frame, similarly to FIGS. 9A and9B.

As illustrated in FIGS. 9A and 9B, in an imaging operation using ageneral CMOS image sensor, local movement vectors (LMV Index=0 to 7)which are detected in each region in a frame are reversed in the sameframe depending on the vertical imaging direction. In contrast to this,in an imaging operation using the global shutter image sensor, a localmovement vector is approximately constant in the same frame asillustrated in FIG. 10, when there is no moving body in the frame.

For this reason, moving image contents which are generated by the globalshutter image sensor have a small difference between the integrationvalue of quantity of motion of the forward direction frame boundary(NormalDiffSum) and the integration value of quantity of motion of thebackward direction frame boundary (InverseDiffSum). For example, thedifference is equal to or smaller than a threshold value. Therefore, thedetermination unit 120 makes a determination on whether the shuttersystem is the global shutter system or the focal plane shutter system(rolling shutter system) using the following conditional expression(formula 8).

$\begin{matrix}{{{Imaging}\mspace{14mu}{Information}} = \left\{ \begin{matrix}{{Global}\mspace{14mu}{Shutter}} & \begin{matrix}{{{{Normal}\mspace{14mu}{Diff}\mspace{14mu}{Sum}} -}} \\{{{{Inverse}\mspace{14mu}{Diff}\mspace{14mu}{Sum}}} \leq {thresh}}\end{matrix} \\\begin{matrix}{{Rolling}\mspace{14mu}{Shutter}\mspace{14mu}{Vertical}} \\{{imaging}\mspace{14mu}{forward}\mspace{14mu}{directon}}\end{matrix} & \begin{matrix}{{{{Normal}\mspace{14mu}{{Dif}f}\mspace{14mu}{Sum}} -}} \\{{{{Inverse}\mspace{14mu}{Diff}\mspace{14mu}{Sum}}} \leq {thresh}} \\{{{AND}\mspace{14mu}{Normal}\mspace{14mu}{Diff}\mspace{14mu}{Sum}} <} \\{{Inverse}\mspace{14mu}{Diff}\mspace{14mu}{Sum}}\end{matrix} \\\begin{matrix}{{Rolling}\mspace{14mu}{Shutter}\mspace{14mu}{Vertical}} \\{{imaging}\mspace{14mu}{backward}\mspace{14mu}{directon}}\end{matrix} & {Otherwise}\end{matrix} \right.} & (8)\end{matrix}$

Here, “thresh” is a threshold value. As the threshold value, forexample, a value which is suitable for making a determination of whetherthe shutter system is the global shutter system or the focal planeshutter system is set. In addition, the threshold value is changedaccording to a determination period of imaging information of movingimage contents.

In addition, since the above described threshold value is set to beunchangeable regardless of the determination period of the imaginginformation of the moving image contents, the determination may be madebased on a value which is normalized in the determination period of theimaging information of the moving image contents. That is, thedetermination of whether the shutter system is the global shutter systemor the focal plane shutter system may be made using the followingconditional expression (formula 9).

$\begin{matrix}{{{Imaging}\mspace{14mu}{Information}} = \left\{ \begin{matrix}{{Global}\mspace{14mu}{Shutter}} & \begin{matrix}{{{{Normal}\mspace{14mu}{Diff}\mspace{14mu}{Ave}} -}} \\{{{{Inverse}\mspace{14mu}{Diff}\mspace{14mu}{Ave}}} \leq {thresh}}\end{matrix} \\\begin{matrix}{{Rolling}\mspace{14mu}{Shutter}\mspace{14mu}{Vertical}} \\{{imaging}\mspace{14mu}{forward}\mspace{14mu}{directon}}\end{matrix} & \begin{matrix}{{{{Normal}\mspace{14mu}{{Dif}f}\mspace{14mu}{Ave}} -}} \\{{{{Inverse}\mspace{14mu}{Diff}\mspace{14mu}{Ave}}} \leq {thresh}} \\{{{AND}\mspace{14mu}{Normal}\mspace{14mu}{Diff}\mspace{14mu}{Ave}} <} \\{{Inverse}\mspace{14mu}{Diff}\mspace{14mu}{Ave}}\end{matrix} \\\begin{matrix}{{Rolling}\mspace{14mu}{Shutter}\mspace{14mu}{Vertical}} \\{{imaging}\mspace{14mu}{backward}\mspace{14mu}{directon}}\end{matrix} & {Otherwise}\end{matrix} \right.} & (9)\end{matrix}$

Here, NormalDiffAve is a value in which the integration value ofquantity of motion of the forward direction frame boundary(NormalDiffSum) is normalized in a determination period of imaginginformation. In addition, InverseDiffAve is a value in which theintegration value of quantity of motion of the backward direction frameboundary (InverseDiffSum) is normalized in the determination period ofthe imaging information.

In addition, these normalized values (NormalDiffAve, InverseDiffAve) canbe obtained by averaging each integration value using the number oftimes of addition of each integration value (NormalDiffSum,InverseDiffSum). That is, the normalized values can be obtained byaveraging (for example, dividing each integration value by the number oftimes) each integration value using the number of times of updating eachintegration value (NormalDiffSum, InverseDiffSum).

Here, when it is determined that photographing was performed using theglobal shutter system as a result of determining the shutter system, thecorrection unit 130 outputs moving image contents as a corrected imagewithout performing correction processing with respect to the movingimage contents.

In this manner, it is possible to accurately determine the shuttersystem of the moving image contents of which imaging information at thetime of imaging operation is not known. That is, it is possible todetermine that moving image contents of which imaging information is notknown at the time of imaging operation are moving image contents whichare imaged using the global shutter image sensor. In this manner, it ispossible to prevent unnecessary correction of focal plane distortionfrom being performed with respect to moving image contents which areimaged using the global shutter image sensor (moving image contents ofwhich imaging operation is performed in an environment which has nofocal plane distortion in principle). That is, it is possible to confirmwhether or not the correction of moving image contents is necessarybased on a determination result, by determining a shutter system.

In this manner, the determination unit 120 can determine imaginginformation at the time of imaging operation of the moving imagecontents based on continuity of local movement vectors between frameswhich are neighboring time sequentially among frames which configure themoving image contents.

In addition, for example, the frames which are neighboring timesequentially among the frames which configure the moving image contentsare set to a first frame and a second frame. In this case, thedetermination unit 120 performs determination based on a comparisonresult of a local movement vector which is obtained from a region on oneend side in a specific direction of the first frame, and a localmovement vector which is obtained from a region on the other end side ina specific direction of the second frame. Here, the specific directionof each frame can be set to a vertical direction of each frame, forexample.

In addition, for example, the local movement vector which is obtainedfrom the region on one end side of the first frame is set to a firstmovement vector, and the local movement vector which is obtained fromthe region on the other end side of the first frame is set to a secondmovement vector. In addition, a local movement vector which is obtainedfrom a region on one end side of the second frame is set to a thirdmovement vector, and a local movement vector which is obtained from aregion on the other end side of the second frame is set to a fourthmovement vector. In this case, the determination unit 120 makes adetermination using a value which is calculated based on a comparisonresult of the first movement vector and the fourth movement vector, anda comparison result of the second movement vector and the third movementvector (for example, value calculated using formulas 3 and 4, orformulas 5 and 6).

In addition, in the embodiment of the present application, an example inwhich an image which is cut out from moving image contents is set to aframe image has been described, however, it is possible to apply theembodiment of the present application to a case in which an image whichis cut out from moving image contents is set to a field image.

Correction Example of Focal Plane Distortion

FIG. 11 is a block diagram which illustrates a functional configurationexample of the correction unit 130 according to the first embodiment ofthe present application.

The correction unit 130 performs appropriate correction processing offocal plane distortion using imaging information which is determined bythe determination unit 120. Specifically, the correction unit 130includes a quantity of displacement estimation unit 131, a correctionamount deriving unit 132, and a correction processing unit 133.

The quantity of displacement estimation unit 131 estimates a quantity ofdisplacement of moving image contents which are recorded in the movingimage contents recording unit 110, and outputs the estimated quantity ofdisplacement to the correction amount deriving unit 132. In addition, amethod of estimating a quantity of displacement will be described indetail with reference to FIGS. 12A to 13D.

The correction amount deriving unit 132 derives a correction amount forcorrecting moving image contents which are recorded in the moving imagecontents recording unit 110 based on a quantity of displacement which isestimated by the quantity of displacement estimation unit 131 (estimatedquantity of displacement) and a determination result using thedetermination unit 120 (imaging information). In addition, thecorrection amount deriving unit 132 outputs the derived correctionamount to the correction processing unit 133. In addition, a method ofderiving a correction amount will be described in detail with referenceto FIGS. 14A to 14C.

The correction processing unit 133 corrects moving image contents whichare recorded in the moving image contents recording unit 110 based on acorrection amount which is derived from the correction amount derivingunit 132, and outputs corrected moving image contents to the output unit140. In addition, the correction method will be described in detail withreference to FIGS. 12A to 14C.

FIGS. 12A to 14C are diagrams which illustrate a relationship among animage which is a target of correction processing using the correctionunit 130 according to the first embodiment of the present applicationand a corrected image, and each piece of information which is used inthe correction processing.

FIGS. 12A to 13D illustrate graphs in which a vertical axis is set to anaxis which denotes each frame in time sequence, and a horizontal axis isset to an axis which denotes each information related to each frame. Inaddition, marks (n−2 to n+3) for specifying frames are denoted on theright side in respective FIGS. 12A to 13D.

In FIG. 12A, a graph which time sequentially denotes a hand shakequantity in the horizontal direction of an image sensor is illustrated.In FIG. 12B, an example of an image which is generated when there is thehand shake quantity illustrated in FIG. 12A is illustrated.

In FIG. 12C, an example of a quantity of displacement which is estimatedwith respect to the image which is illustrated in FIG. 12B (quantity ofdisplacement of image sensor at time of imaging operation) isillustrated.

In FIG. 12D, an example of a correction amount which is derived based ona quantity of displacement (quantity of displacement of image sensor attime of imaging operation) which is illustrated in FIG. 12C, and adetermination result using the determination unit 120 is illustrated.

In FIG. 12E, an example of a corrected image when the image illustratedin FIG. 12B is corrected based on the correction amount illustrated inFIG. 12D is illustrated.

For example, a case in which there is hand shake of the hand shakeamount illustrated in FIG. 12A (hand shake in the horizontal direction),and the imaging information illustrated in FIG. 12B is generated isassumed. In addition, a relationship among a motion of the CMOS imagesensor, the vertical imaging direction, and the imaged image is the sameas that illustrated in FIG. 6.

Here, the hand shake illustrated in FIG. 12A is a non-linear motion. Inthis manner, when the hand shake is the non-linear motion, non-lineardistortion also occurs in the imaged image, as illustrated in FIG. 12B.

Estimation Example of Quantity of Displacement

Here, in FIG. 12A, the hand shake amount of the image sensor in thehorizontal direction is time sequentially denoted, however, in an actualsystem, it is difficult to obtain information on a hand shake amount ineach line from moving image contents, in general. For this reason, it isnecessary to estimate a quantity of displacement of the image sensor ata time of imaging operation of an imaged image, based on the imagedimage.

For example, the quantity of displacement estimation unit 131 derives anestimated quantity of displacement (estimated quantity of displacementof image sensor at time of imaging operation) which is illustrated inFIG. 12C based on the imaged image illustrated in FIG. 12B. Theestimated quantity of displacement is a value which is calculated basedon estimation on how much the image sensor has moved in one frameperiod. In addition, an example of deriving one estimated quantity ofdisplacement in one frame period is illustrated in FIG. 12C, however, aplurality of estimated quantities of displacement may be derived in oneframe period.

Here, an estimation method of a quantity of displacement of the imagesensor at the time of imaging operation using the quantity ofdisplacement estimation unit 131 will be described with reference toFIGS. 13A to 13D. In addition, in the example illustrated in FIGS. 13Ato 13D, for ease of description, an example is illustrated in which LMVsare calculated from four regions with respect to each frame, and anestimated quantity of displacement is calculated based on the LMVs.

The imaged image which is illustrated in FIG. 13A corresponds to theimaged image which is illustrated in FIG. 12B. In addition, it is setsuch that the LMVs which are illustrated in FIG. 13B (LMV Index=0 to 3)are obtained in each frame with respect to the imaged image which isillustrated in FIG. 13A.

Here, as a method of estimating a quantity of displacement of an imagesensor, for example, it is possible to use a method in which a GlobalMotion Vector (GMV) which is calculated based on an LMV is set to anestimation amount. This estimation method is the simplest method, and itis possible to obtain Estimated Displacement in Horizontal direction(EDH) as illustrated in the following formula 10. In addition, thesuffix n is a serial number for identifying each frame.

$\begin{matrix}{{EDH}_{n} = {{GMV}_{n} = \frac{\sum\limits_{i = 0}^{3}\;{{LMV}\lbrack i\rbrack}_{n}}{4}}} & (10)\end{matrix}$

In FIG. 13C, an example of estimated quantity of displacement obtainedusing formula 10 is illustrated.

Here, the LMV is calculated based on a difference between positioncoordinates of the current frame (for example, nth frame) and positioncoordinates of the previous frame (for example, n−1th frame) withrespect to a certain reference image. For this reason, the LMV in thecurrent frame is a quantity of displacement in a period from theprevious frame to the current frame. In contrast to this, distortion ofthe imaged image in the current frame occurs based on the quantity ofdisplacement of the current frame.

For this reason, as described above, when the GMV is set to theestimated quantity of displacement of the image sensor, there is a casein which a phase shift between the LMV and a waveform of hand shakeoccurs.

For example, in the imaged image of the nth frame, and in the imagedimage of the n−1th frame illustrated in FIG. 13A, distortion occurs atthe same angle from the upper left to the lower right. However, asillustrated in FIG. 13C, there is a large difference in the estimatedquantity of displacement which is obtained based on the GMV between thenth frame and the n−1th frame.

Therefore, in order to alleviate such a phase shift, the estimatedquantity of displacement (EDH) may be obtained using the followingformula 11.

$\begin{matrix}{{EDH}_{n} = \frac{\left( {{\sum\limits_{i = 0}^{3}\;{{LMV}\lbrack i\rbrack}_{n}} + {\sum\limits_{i = 0}^{3}\;{{LMV}\lbrack i\rbrack}_{n + 1}}} \right)}{8}} & (11)\end{matrix}$

That is, it is possible to obtain the estimated quantity of displacementof the current frame based on all of LMVs of the subsequent frame to thecurrent frame (for example, n+1th frame), and all of LMVs of the currentframe (for example, nth frame). In this manner, it is possible toalleviate the phase shift.

In FIG. 13D, an example of an estimated quantity of displacement whichis obtained using formula 11 is illustrated.

In addition, it may be possible to obtain a quantity of estimation usinga center of a difference, using only the LMVs which are included in atarget frame period. For example, the estimated quantity of displacement(EDH) may be obtained using the following formula 12.

$\begin{matrix}{{EDH}_{n} = \frac{\left( {{\sum\limits_{i = 2}^{3}\;{{LMV}\lbrack i\rbrack}_{n}} + {\sum\limits_{i = 0}^{1}\;{{LMV}\lbrack i\rbrack}_{n + 1}}} \right)}{4}} & (12)\end{matrix}$

Example of Deriving Correction Amount

The correction amount deriving unit 132 derives a correction amount ineach line in each frame period illustrated in FIG. 12D based on anestimated quantity of displacement of the image sensor illustrated inFIG. 12C, and the imaging information which is output from thedetermination unit 120.

Here, a method of deriving a correction amount using the correctionamount deriving unit 132 will be described in detail.

In addition, as the correction method, for example, there are a linearcorrection in which each line is corrected using a constant correctionamount, and a non-linear correction in which each line is correctedusing a non-constant correction amount in a frame period of a correctiontarget. Here, a calculation method of a correction amount when thecorrection is performed using the linear correction will be describedwith reference to FIGS. 14A to 14C.

As illustrated in FIG. 14A, in imaging using a general image sensor, apredetermined time interval (denoted by Vertical Blanking Interval (VBI)in FIG. 14A) is present between each of frames. Therefore, it ispossible to obtain a position correction amount in the horizontaldirection (Correction Value for Horizontal Displacement (CVHD)) of acertain line (for example, kth line) of a certain frame (for example,nth frame) using the following formula 13, by taking the time interval(VBI) into consideration.

$\begin{matrix}{{CVDH}_{n,k} = {{CFSP} \cdot {EDH}_{n} \cdot \frac{{1/{FrameRatio}} - {VBI}}{1/{FrameRatio}} \cdot \left( {\frac{k}{{LineNum} - 1} - A} \right)}} & (13)\end{matrix}$

Here, LineNum is a value which denotes the number of imaging lines inone frame. In addition, FrameRatio is a value which denotes the numberof imaged sheets per second. In addition, Correction Factor for SensorProperties (CFSP) is a correction coefficient with respect to an imagedimage, and a value (correction coefficient) which is determined based onthe following conditional expression (formula 14). That is, CFSP isdetermined based on imaging information corresponding to the abovedescribed conditional expression (formula 9).

$\begin{matrix}{{CFSP} = \left\{ \begin{matrix}{0\mspace{14mu}{Imaging}\mspace{14mu}{{Information}:}} & {{Global}\mspace{14mu}{Shutter}} \\{1\mspace{14mu}{Imaging}\mspace{14mu}{{Information}:}} & {{Vertical}\mspace{14mu}{imaging}\mspace{14mu}{forward}\mspace{14mu}{directon}} \\{{- 1}\mspace{14mu}{Imaging}\mspace{14mu}{{Information}:}} & {{Vertical}\mspace{14mu}{imaging}\mspace{14mu}{backward}\mspace{14mu}{directon}}\end{matrix} \right.} & (14)\end{matrix}$

In addition, A is a correction-centered adjustment coefficient. Forexample, when setting A=0.5, A becomes a correction amount in which acenter line in an imaging region is set to a center.

For example, a case is assumed in which an estimated quantity ofdisplacement which is derived from the quantity of displacementestimation unit 131 is set to an estimated quantity of displacementwhich is illustrated in FIG. 14B. In this case, the correction amountderiving unit 132 calculates a correction amount using formula 13 asillustrated in FIG. 14C.

In addition, the correction processing unit 133 creates a correctedimage by performing a correction with respect to an imaged image basedon a correction amount which is derived from the correction amountderiving unit 132. For example, the correction processing unit 133creates the corrected image which is illustrated in FIG. 12E withrespect to the imaged image illustrated in FIG. 12B based on acorrection amount in each line illustrated in FIG. 12D.

In this manner, the correction unit 130 performs a correction of focalplane distortion with respect to moving image contents based on animaging direction which is determined by the determination unit 120,when it is determined that the shutter system is the focal plane shuttersystem using the determination unit 120. On the other hand, thecorrection unit 130 does not perform the correction of focal planedistortion with respect to moving image contents when it is determinedthat the shutter system is the global shutter system using thedetermination unit 120.

In FIGS. 12A to 12E and FIGS. 14A to 14C, examples in which a correctionis performed by calculating a linear correction amount in one frameperiod have been illustrated, however, the correction may be performedby calculating a non-linear correction amount.

In addition, hitherto, a correction in the horizontal direction has beendescribed, however, it is possible to perform a correction with respectto the vertical direction using the same method. That is, in thecorrection in the horizontal direction, displacement in the horizontaldirection has been corrected in each line, however, in the correction inthe vertical direction, displacement (expansion and contraction) in thevertical direction is corrected in each line.

In this manner, it is possible to perform appropriate correctionprocessing with respect to moving image contents, and to perform astable correction.

Operation Example of Image Processing Device

FIG. 15 is a flowchart which illustrates an example of processing orderof correction processing using the image processing device 100 accordingto the first embodiment of the present application. In addition, in FIG.15, an example in which a vertical imaging direction is determined asimaging information is illustrated.

First, the determination unit 120 initializes a reference frame index,the integration value of quantity of motion of the forward directionframe boundary, and the integration value of quantity of motion of thebackward direction frame boundary (Step S901). Here, the reference frameindex is, for example, an index for specifying a frame which is aprocessing target. For example, the determination unit 120 initializesthe reference frame index so as to indicate a top frame which configuresmoving image contents which are recorded in the moving image contentsrecording unit 110. In addition, for example, the determination unit 120initializes the respective integration value of quantity of motion ofthe forward direction frame boundary and integration value of quantityof motion of the backward direction frame boundary using zero.

Subsequently, the determination unit 120 obtains a frame which isindicated by the reference frame index from the moving image contentswhich are recorded in the moving image contents recording unit 110, andobtains a local movement vector from each region in the frame (StepS902). For example, as illustrated in FIGS. 8B and 8C, local movementvectors are obtained from eight regions in a frame.

Subsequently, the determination unit 120 calculates the quantity ofmotion of the forward direction frame boundary, and the quantity ofmotion of the backward direction frame boundary using the obtained localmovement vectors (Step S903). That is, the determination unit 120calculates the quantity of motion of the forward direction frameboundary (NormalFrameDiff_(n)) using the above described formula 1, andcalculates the quantity of motion of the backward direction frameboundary (InverseFrameDiff_(n)) using the above described formula 2.

Subsequently, the determination unit 120 calculates the integrationvalue of quantity of motion of the forward direction frame boundary, andthe integration value of quantity of motion of the backward directionframe boundary by adding the respective quantity of motion of theforward direction frame boundary and quantity of motion of the backwarddirection frame boundary to each integration value (Step S904). That is,the determination unit 120 calculates the integration value of quantityof motion of the forward direction frame boundary (NormalDiffSum) usingthe above described formula 3, and calculates the integration value ofquantity of motion of the backward direction frame boundary(InverseDiffSum) using the above described formula 4. In addition, asdescribed above, when a local movement vector is not detected withrespect to the top frame, each integration value may be calculated usingthe above described formulas 5 and 6.

Subsequently, the determination unit 120 updates the reference frameindex by updating the index as much as a regulated amount (Step S905).For example, a value of the reference frame index is incremented, and 1is added.

Subsequently, the determination unit 120 checks the reference frameindex, and determines whether or not a period of the reference frame isended (Step S906). For example, the determination unit 120 determineswhether or not a frame which is indicated by the reference frame indexis a rear end frame of the moving image contents which are recorded inthe moving image contents recording unit 110. In addition, when theframe which is indicated by the reference frame index is the rear endframe, the determination unit 120 determines that the period of thereference frame is ended.

When the period of the reference frame is not ended (Step S906), theprocess returns to Step S902. On the other hand, when the period ofreference frame is ended (Step S906), the determination unit 120performs determination on imaging information using the integrationvalue of quantity of motion of the forward direction frame boundary, andthe integration value of quantity of motion of the backward directionframe boundary (Step S907). That is, the determination unit 120 makes adetermination using the above described determination condition (formula7), and obtains the vertical imaging direction as the imaginginformation. In addition, Step S907 is an example of a determinationprocedure which is described in claims.

Subsequently, the correction unit 130 performs correction processing offocal plane distortion with respect to the moving image contents whichare recorded in the moving image contents recording unit 110 based on adetermination result using the determination unit 120 (Step S908).

In addition, when determining a shutter system of the image sensor(global shutter system, focal plane shutter system) as the imaginginformation, the determination is made along with the vertical imagingdirection in Step S907. In this case, when it is determined that theshutter system of the image sensor is the global shutter system, thecorrection processing of focal plane distortion in Step S908 is omitted.

2. Modification Example

In the first embodiment of the present application, an example in whicha quantity of displacement of an image sensor at a time of imagingoperation is estimated using the quantity of displacement estimationunit 131 has been described. Here, an imaging device on which a sensorfor detecting a posture of a device (for example, gyro sensor) ismounted is present. When an imaging operation is performed using theimaging device, for example, it is considered that an amount of handshake in each line at the time of imaging operation is obtained usingthe sensor, and information related to the obtained amount of hand shakeis recorded by being included in meta information of moving imagecontents. In this manner, it is possible to use information related tothe amount of hand shake at the time of image processing, by recordingthe information related to the amount of hand shake in each line at thetime of imaging operation in the meta information of the moving imagecontents. In this case, it is possible to omit the quantity ofdisplacement estimation unit 131.

Therefore, in the modification example according to the embodiment ofthe present application, an example in which a correction amount isderived using the meta information (information on amount of hand shakein each line at time of imaging operation) of the moving image contentsis illustrated.

Configuration Example of Correction Unit

FIG. 16 is a block diagram which illustrates a functional configurationexample of an image processing device 400 in the modification exampleaccording to the embodiment of the present application. In addition, inthe image processing device 400, a part of the image processing device100 illustrated in FIG. 1 is modified, and only the correction unit 130illustrated in FIG. 1 is different. For this reason, in FIG. 16, only apart of configurations of the image processing device 400 (configurationcorresponding to FIG. 11) is illustrated, and illustrations of otherconfigurations are omitted. In addition, common portions to the imageprocessing device 100 are given the same reference numerals, anddescriptions of a part of these are omitted.

The image processing device 400 includes a moving image contentsrecording unit 110, a determination unit 120, a correction unit 410, andan output unit 140. In addition, the correction unit 410 includes acorrection amount deriving unit 411 and a correction processing unit133. That is, in the correction unit 410, the quantity of displacementestimation unit 131 in the correction unit 130 illustrated in FIG. 11 isomitted.

In addition, it is assumed that information related to an amount of handshake in each line at the time of imaging operation is recorded inmoving image contents which are recorded in the moving image contentsrecording unit 110 as meta information by being correlated.

The correction amount deriving unit 411 derives a correction amount ineach line of each frame period based on the meta information(information on amount of hand shake in each line at time of imagingoperation) of the moving image contents which are recorded in the movingimage contents recording unit 110, and a determination result (imaginginformation) using the determination unit 120. The same deriving exampleof the correction amount as that in the first embodiment of the presentapplication is applied except for a point that the amount of hand shakein each line at the time of imaging operation is used instead of anestimated quantity of displacement.

CONCLUSION

In this manner, according to the embodiment of the present application,it is possible to obtain imaging information of an image sensor at atime of imaging operation (vertical imaging direction) by appropriatelydetermining the imaging information, even in a case of moving imagecontents of which imaging information at the time of imaging operationis not known. It is possible to prevent a correction of focal planedistortion due to a correction using a wrong imaging direction fromfailing, by performing a correction of focal plane distortion using theimaging information (vertical imaging direction) which is obtained inthis manner. In this manner, it is possible to obtain high stability,and to perform a correction of focal plane distortion with highprecision. That is, it is possible to perform an appropriate correctionof focal plane distortion in which imaging information at the time ofimaging operation is appropriately reflected.

That is, according to the embodiment of the present application, it ispossible to execute a robust correction (correction of focal planedistortion) which attenuates hand shake, and distortion of an imagedimage due to a focal plane phenomenon which occurs in a CMOS imagesensor, or the like. In addition, it is possible to appropriatelydetermine an imaging direction of an image sensor at the time of imagingoperation which is necessary in the correction.

In addition, according to the embodiment of the present application, itis possible to obtain imaging information (shutter system of imagesensor) of the image sensor at the time of imaging operation byappropriately determining the imaging information, even in moving imagecontents of which imaging information at the time of imaging operationis not known. In this manner, even when the moving image contents ofwhich the imaging information at the time of imaging operation is notknown are imaged using a CCD image sensor in which focal planedistortion does not occur in principle, it is possible to prevent thecorrection from being erroneously performed.

In this manner, according to the embodiment of the present application,it is possible to appropriately perform correction processing such as acorrection of the focal plane distortion with respect to moving imagecontents of which imaging information at the time of imaging operationis not known in a cloud environment, or the like, for example. Inaddition, it is possible to appropriately perform correction processingsuch as the correction of the focal plane distortion with respect tomoving image contents of which imaging information at the time ofimaging operation is not known, in moving image editing software, movingimage reproducing software, or the like, for example.

In addition, in the embodiment of the present application, the imageprocessing devices 100 and 400 which are integrally configured have beendescribed as examples. However, it is also possible to apply theembodiment of the present application to an image processing system inwhich each function of the image processing devices 100 and 400 isexecuted using a plurality of devices. For example, it is also possibleto apply the embodiment of the present application to an imageprocessing system in which a plurality of devices corresponding to themoving image contents recording unit 110, the determination unit 120,the correction units 130 and 410, and the output unit 140 are connectedusing a network. In addition, the network is, for example, a networksuch as a telephone network, the Internet (for example, public linenetwork), or a wired network or a coaxial cable.

In addition, the above described embodiment is an example for embodyingthe present application, and matters in the embodiment and matters usedto define the present application in claims have correspondence,respectively. Similarly, the matters used to define the presentapplication in claims and the matters in the embodiment of the presentapplication to which the same name is attached have correspondence,respectively. However, the present application is not limited to theembodiment, and can be embodied by performing various modifications withrespect to the embodiment without departing from the scope of thepresent application.

In addition, the processing procedure which is described in the abovedescribed embodiment may be treated as a method having a series of theseprocedures, and may be treated as a program for executing the series ofprocedures in a computer, and a recording medium which records theprogram. As the recording medium, it is possible to use a Compact Disc(CD), a MiniDisc (MD), a Digital Versatile Disc (DVD), a memory card, aBlu-ray (registered trademark) Disc, or the like, for example.

In addition, effects which are described in the specification are merelyexamples, are not limited, or may be another effect.

In addition, the present application can also be configured as follows.

(1) An image processing device which includes a determination unit whichdetermines a scheme of capturing a pixel value of an imaging element ata time of imaging operation of moving image contents using a featureamount which is obtained from a plurality of regions in a frame whichconfigures the moving image contents.

(2) The image processing device described in (1), in which thedetermination unit determines at least one of an imaging direction and ashutter system of the imaging element at the time of imaging operation,as the capturing scheme.

(3) The image processing device described in (1) or (2), in which thedetermination unit makes the determination based on continuity of localmovement vectors between frames which are neighboring time sequentiallyamong frames which configure the moving image contents, by obtaining thelocal movement vector in each of the plurality of regions as the featureamount.

(4) The image processing device described in any one of (1) to (3), inwhich the determination unit obtains a local movement vector for each ofthe plurality of regions as the feature amount, sets frames which areneighboring time sequentially among frames which configure the movingimage contents to a first frame and a second frame, and makes thedetermination based on a comparison result of a local movement vectorwhich is obtained from a region on one end side in a specific directionof the first frame, and a local movement vector which is obtained from aregion on the other end side in the specific direction of the secondframe.

(5) The image processing device described in (4), in which thedetermination unit makes the determination using a value which iscalculated based on a comparison result of a first movement vector and afourth movement vector, and a comparison result of a second movementvector and a third movement vector the first movement vector being thelocal movement vector obtained from a region on the one end side of thefirst frame, the second movement vector being the local movement vectorobtained from the region on the other end side of the first frame, thethird movement vector being the local movement vector obtained from aregion on the one end side of the second frame, and the fourth movementvector being the local movement vector obtained from a region on theother end side of the second frame.

(6) The image processing device described in any one of (1) to (5), inwhich a correction unit which performs a correction of focal planedistortion with respect to the moving image contents based on thedetermined capturing scheme is further included.

(7) The image processing device described in (6), in which thedetermination unit determines at least one of the imaging direction andthe shutter system of the imaging element at the time of imagingoperation as the capturing scheme, and the correction unit performs thecorrection of the focal plane distortion with respect to the movingimage contents based on the determined imaging direction when it isdetermined that the shutter system is a focal plane shutter system bythe determination unit, and does not perform the correction of the focalplane distortion with respect to the moving image contents when it isdetermined that the shutter system is a global shutter system by thedetermination unit.

(8) The image processing device described in any one of (1) to (7), inwhich the determination unit obtains a feature amount in each of theplurality of regions based on a comparison result of a plurality ofregions in a target frame which configures the moving image contents andanother frame.

(9) An image processing method in which a scheme of capturing a pixelvalue of an imaging element at a time of imaging operation of movingimage contents is determined using a feature amount which is obtainedfrom a plurality of regions in a frame which configures the moving imagecontents.

(10) A program which causes a computer to execute a determinationprocedure which determines a scheme of capturing a pixel value of animaging element at a time of imaging operation of moving image contentsusing a feature amount which is obtained from a plurality of regions ina frame which configures the moving image contents.

It should be understood that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present subjectmatter and without diminishing its intended advantages. It is thereforeintended that such changes and modifications be covered by the appendedclaims.

The invention is claimed as follows:
 1. An image processing devicecomprising: a determination unit which determines a scheme of capturinga pixel value of an imaging element at a time of imaging operation ofmoving image contents using a feature amount which is obtained from aplurality of regions in a frame which configures the moving imagecontents, wherein the determination unit obtains a local movement vectorfor each of the plurality of regions as the feature amount, sets frameswhich are neighboring time sequentially among frames which configure themoving image contents to a first frame and a second frame, and makesdetermination based on a comparison result of a local movement vectorwhich is obtained from a region on one end side in a specific directionof the first frame, and a local movement vector which is obtained from aregion on the other end side in the specific direction of the secondframe.
 2. The image processing device according to claim 1, wherein thedetermination unit determines at least one of an imaging direction and ashutter system of the imaging element at the time of imaging operation,as the capturing scheme.
 3. The image processing device according toclaim 1, wherein the determination unit makes the determination based oncontinuity of local movement vectors between frames which areneighboring time sequentially among frames which configure the movingimage contents, by obtaining the local movement vector in each of theplurality of regions as the feature amount.
 4. The image processingdevice according to claim 1, wherein the determination unit makes thedetermination using a value which is calculated based on a comparisonresult of a first movement vector and a fourth movement vector, and acomparison result of a second movement vector and a third movementvector, the first movement vector being the local movement vectorobtained from a region on the one end side of the first frame, thesecond movement vector being the local movement vector obtained from theregion on the other end side of the first frame, the third movementvector being the local movement vector obtained from a region on the oneend side of the second frame, and the fourth movement vector being thelocal movement vector obtained from a region on the other end side ofthe second frame.
 5. The image processing device according to claim 1,further comprising: a correction unit which performs a correction offocal plane distortion with respect to the moving image contents basedon the determined capturing scheme.
 6. The image processing deviceaccording to claim 4, wherein the determination unit determines at leastone of the imaging direction and the shutter system of the imagingelement at the time of imaging operation as the capturing scheme, andwherein the correction unit performs the correction of the focal planedistortion with respect to the moving image contents based on thedetermined imaging direction when it is determined that the shuttersystem is a focal plane shutter system by the determination unit, anddoes not perform the correction of the focal plane distortion withrespect to the moving image contents when it is determined that theshutter system is a global shutter system by the determination unit. 7.The image processing device according to claim 1, wherein thedetermination unit obtains a feature amount in each of the plurality ofregions based on a comparison result of a plurality of regions in atarget frame which configures the moving image contents and anotherframe.
 8. An image processing method comprising: determining a scheme ofcapturing a pixel value of an imaging element at a time of imagingoperation of moving image contents using a feature amount which isobtained from a plurality of regions in a frame which configures themoving image contents, obtaining a local movement vector for each of theplurality of regions as the feature amount, setting frames which areneighboring time sequentially among frames which configure the movingimage contents to a first frame and a second frame, and makingdetermination based on a comparison result of a local movement vectorwhich is obtained from a region on one end side in a specific directionof the first frame, and a local movement vector which is obtained from aregion on the other end side in the specific direction of the secondframe.
 9. A non-transitory storage medium storing a program which, whenexecuted by a computer, causes the computer to execute: a determinationprocedure of determining a scheme of capturing a pixel value of animaging element at a time of imaging operation of moving image contentsusing a feature amount which is obtained from a plurality of regions ina frame which configures the moving image contents, obtaining a localmovement vector for each of the plurality of regions as the featureamount, setting frames which are neighboring time sequentially amongframes which configure the moving image contents to a first frame and asecond frame, and making determination based on a comparison result of alocal movement vector which is obtained from a region on one end side ina specific direction of the first frame, and a local movement vectorwhich is obtained from a region on the other end side in the specificdirection of the second frame.
 10. An image processing devicecomprising: a determination unit which determines a scheme of capturinga pixel value of an imaging element at a time of imaging operation ofmoving image contents using a feature amount which is obtained from aplurality of regions in a frame which configures the moving imagecontents, and a correction unit which performs a correction of focalplane distortion with respect to the moving image contents based on thedetermined capturing scheme, wherein the determination unit determinesat least one of the imaging direction and the shutter system of theimaging element at the time of imaging operation as the capturingscheme, and wherein the correction unit performs the correction of thefocal plane distortion with respect to the moving image contents basedon the determined imaging direction when it is determined that theshutter system is a focal plane shutter system by the determinationunit, and does not perform the correction of the focal plane distortionwith respect to the moving image contents when it is determined that theshutter system is a global shutter system by the determination unit. 11.An image processing method comprising: determining, by a determinationunit, a scheme of capturing a pixel value of an imaging element at atime of imaging operation of moving image contents using a featureamount which is obtained from a plurality of regions in a frame whichconfigures the moving image contents, and performing, by a correctionunit, a correction of focal plane distortion with respect to the movingimage contents based on the determined capturing scheme, wherein thedetermination unit determines at least one of the imaging direction andthe shutter system of the imaging element at the time of imagingoperation as the capturing scheme, and wherein the correction unitperforms the correction of the focal plane distortion with respect tothe moving image contents based on the determined imaging direction whenit is determined that the shutter system is a focal plane shutter systemby the determination unit, and does not perform the correction of thefocal plane distortion with respect to the moving image contents when itis determined that the shutter system is a global shutter system by thedetermination unit.
 12. A non-transitory storage medium storing aprogram which, when executed by a computer, causes the computer toexecute: a determination procedure, by a determination unit, ofdetermining a scheme of capturing a pixel value of an imaging element ata time of imaging operation of moving image contents using a featureamount which is obtained from a plurality of regions in a frame whichconfigures the moving image contents, and a correction procedure, by acorrection unit, of correcting focal plane distortion with respect tothe moving image contents based on the determined capturing scheme,wherein the determination unit determines at least one of the imagingdirection and the shutter system of the imaging element at the time ofimaging operation as the capturing scheme, and wherein the correctionunit performs the correction of the focal plane distortion with respectto the moving image contents based on the determined imaging directionwhen it is determined that the shutter system is a focal plane shuttersystem by the determination unit, and does not perform the correction ofthe focal plane distortion with respect to the moving image contentswhen it is determined that the shutter system is a global shutter systemby the determination unit.
 13. The image processing device according toclaim 10, wherein the determination unit determines at least one of animaging direction and a shutter system of the imaging element at thetime of imaging operation, as the capturing scheme.
 14. The imageprocessing device according to claim 10, wherein the determination unitmakes the determination based on continuity of local movement vectorsbetween frames which are neighboring time sequentially among frameswhich configure the moving image contents, by obtaining the localmovement vector in each of the plurality of regions as the featureamount.
 15. The image processing device according to claim 10, whereinthe determination unit obtains a feature amount in each of the pluralityof regions based on a comparison result of a plurality of regions in atarget frame which configures the moving image contents and anotherframe.